Define clean-install HA qualification by ankitgoswami · Pull Request #889 · block/proto-fleet

ankitgoswami · 2026-08-07T18:27:25Z

Reviewable diff: +216/-20 across 3 files (excludes generated, test, and story files).

Summary

Defines the evidence required before the fixed three-host Fleet application, database, DCS, and VIP profile is called supported at a recorded fleet scale and topology. The packaged release must pass the complete procedure separately on Debian arm64 Raspberry Pi hardware, Ubuntu amd64, and 64-bit Raspberry Pi OS arm64. Other derivatives remain installer-compatible but unqualified; update qualification and deferred Fleet Node, scheduler-recovery, and alert-delivery behavior are separate.

Stack: #887 -> #888 -> #889 -> #890 -> #891 -> #892. This documentation-only diff is relative to installer PR #888. Update qualification is deferred to #892.

How it works

For each required platform, operators install the same packaged release concurrently on three clean hosts and record the release bundle and deployment-manifest digests, container image identities, host-package versions, host hardware, miner count, plugin mix, connection topology, command backlog, and timing evidence. Results from different platforms cannot be combined into one passing report.

The checklist verifies reboot persistence, independent and combined Fleet/database host loss, acknowledged-write durability, direct host-pinned writer exclusivity, passive request rejection, demotion cancellation, observable firewall behavior, stale command completion rejection, an active curtailment target through failover, repeated failovers, and a 24-hour soak. Every injected Fleet, PostgreSQL, etcd, or VIP fault must also continue or resume MQTT curtailment and reach exact shed and restoration targets within 180 seconds. A controller-side command ledger requires every acknowledged PENDING command to succeed within 60 seconds and permits interruption failure only for commands proven PROCESSING when the fault began. Controller-clock probe windows conservatively bound possible VIP ownership, while a fleet-wide external recorder captures every command, plugin, and curtailment request independently of sampled health. Any evidence gap, overlapping old/new-holder work, or failed gate leaves that exact artifact and host profile unsupported.

flowchart LR
  P["Select one required platform"] --> I["Clean install"]
  I --> R["Reboot recovery"]
  R --> F["Failure matrix"]
  F --> C["Commands and curtailment"]
  C --> S["Repeated failover and soak"]
  S --> V{"Every gate passes?"}
  V -->|yes| N["Record this platform as qualified"]
  V -->|no| U["Keep this platform unsupported"]

Areas of the code involved

Area / package / file	What changed	Why it matters for review
`deployment-files/ha/QUALIFICATION.md`	Adds the redacted qualification checklist, required platform matrix, and result tables	Check that every support claim requires durable platform-specific evidence
`deployment-files/ha/README.md`	Distinguishes installer-compatible targets from qualified hardware profiles	Check that support claims require recorded evidence
`docs/rfcs/0002-active-passive-fleet-ha.md`	Aligns the initial support contract with deferred Fleet Node, scheduler-recovery, and alert-delivery work	Check that local degraded status remains required

Key technical decisions & trade-offs

Qualification uses a packaged release rather than a source checkout.
Each required operating-system, architecture, and hardware combination gets its own complete report; passing one target does not make an architecture-wide claim.
Support is limited to the exact artifact identities, host-package versions, recorded host hardware, fleet scale and topology, and an HA segment restricted to the three hosts and trusted network infrastructure.
PROCESSING-command recovery covers server state; exactly-once device effects and device-side fencing remain outside the profile's support claim.
Every injected Fleet, PostgreSQL, etcd, or VIP fault carries an active curtailment target and separate 180-second shed and restoration bounds.
The address recorder expands each possible ownership interval across delivery uncertainty; ambiguous overlap fails rather than relying on event arrival order.
Fleet Node reconnect scale, scheduler recovery, and alert delivery remain deferred.
The procedure records outcomes and timings without adding test-only runtime behavior.

Testing & validation

The static HA profile check and diff check pass.
Real Debian arm64 Raspberry Pi, Ubuntu amd64, and Raspberry Pi OS arm64 runs are pending.
Hardware failovers, the 180-second curtailment target, repetition counts, and the 24-hour soak are pending, so the qualification verdict remains pending.

github-actions · 2026-08-07T18:29:46Z

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

Reviewed pull request diff only (daf2af34837015b0c55a0c2597e611e6b6940da5...9dc0e101ffe66ae37c015b42f6834bed2b0710e8, exact PR three-dot diff)

Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

Review Summary

Overall Risk: MEDIUM

The new HA qualification procedure contains contradictory and insufficiently isolated test instructions. These defects could either prevent qualification entirely or allow important fencing paths to remain untested.

Findings

[MEDIUM] Qualification uses an unsupported installation flow

Category: Infrastructure
Location: deployment-files/ha/QUALIFICATION.md:61
Description: The procedure instructs operators to copy raw node.env and secret directories and run fleet-ha install concurrently on all hosts. The supported installer instead initializes ha-a first, generates checksummed role-specific bundles, and requires ha-b and ha-c to receive those bundles as arguments. Running without a bundle makes each host attempt cluster initialization as ha-a.
Impact: A literal qualification run can fail to form a cluster or create incompatible roles and credentials. Ad-hoc workarounds may also bypass the bundle checksum and release-identity validation that the guided flow provides.
Recommendation: Replace these steps with the exact README workflow: initialize ha-a, copy and verify its generated bundles, then install each peer using its matching host bundle. Do not instruct operators to stage raw environment or secret directories.

[MEDIUM] Failure gates do not isolate the fencing path being qualified

Category: Reliability
Location: deployment-files/ha/QUALIFICATION.md:124
Description: The DCS, database, and VIP/interface gates do not require non-target paths to remain healthy. Because all services use the same host LAN interface, bringing that interface down can satisfy every gate through combined DCS, database, and endpoint loss without exercising each independent fencing signal.
Impact: A broken endpoint-heartbeat or DCS-specific fencing path could still be marked qualified. For example, loss of keepalived alone could leave the active lease held and blackhole control traffic indefinitely, even though a broad interface-loss test passes.
Recommendation: Specify selective fault injection for each row and continuously assert that non-target DCS, database, and endpoint paths remain healthy until the expected fencing action occurs. Test complete interface loss separately as a compound failure.

[MEDIUM] The no-overlap requirement contradicts the stalled plugin-call gate

Category: Reliability
Location: deployment-files/ha/QUALIFICATION.md:136
Description: The procedure requires no old-holder request to remain in flight when new-holder work begins, but this gate deliberately SIGSTOPs an active process with a stalled plugin call, allows the peer to recover, and only later resumes the old process to receive its queued result. Under the document's acceptance-to-response definition, that old call remains in flight during takeover. The document also explicitly excludes device-side fencing.
Impact: A conforming implementation cannot satisfy all stated gates, encouraging inconsistent interpretation or unsupported PASS results. The ambiguity can also conceal overlapping or duplicate device effects during failover.
Recommendation: Limit the software guarantee to rejecting stale database transitions and preventing new dispatches from the old holder after lease loss. Record already-issued device effects as ambiguous. If strict effect non-overlap is required, add a device-side fencing or acknowledgement protocol before permitting takeover.

Notes

The scoped diff changes documentation only. It introduces no direct authentication, SQL, protobuf, pool-address, or credential-handling code changes.

_{Generated by Codex Security Review |

Triggered by: @ankitgoswami |

Review workflow run}

github-actions Bot assigned ankitgoswami Aug 7, 2026

github-actions Bot added documentation Improvements or additions to documentation review-policy: needs-review Managed by the Review Policy workflow. labels Aug 7, 2026

This was referenced Aug 7, 2026

Add local HA operator status #887

Merged

Guide HA installation on supported apt-based hosts #888

Open

ankitgoswami changed the title ~~test(ha): define clean install qualification~~ Define clean-install HA qualification Aug 7, 2026

This was referenced Aug 7, 2026

Update passive HA application hosts #890

Open

Complete HA updates through bounded failover #891

Open

Define adjacent-release HA update qualification #892

Draft

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch from be18ad0 to 1e137c7 Compare August 7, 2026 18:44

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch 2 times, most recently from ea992c4 to eed7b91 Compare August 7, 2026 19:11

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch 2 times, most recently from 1450e54 to dece2e5 Compare August 7, 2026 19:36

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch 2 times, most recently from 305bce3 to 664159b Compare August 7, 2026 19:47

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch from 664159b to fd51051 Compare August 7, 2026 19:52

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch 2 times, most recently from 4f4fb38 to d567ef7 Compare August 7, 2026 20:11

ankitgoswami force-pushed the ankitg/ha-clean-install-qualification branch from d567ef7 to 788fbde Compare August 7, 2026 20:19

ankitgoswami added 30 commits August 10, 2026 18:54

Correct debugger firewall qualification

00ce777

Scope HA soak evidence precisely

dc31df3

Simplify HA qualification gates

590be6d

Strengthen HA qualification gates

bd997c2

Tighten HA qualification evidence

042be15

Clarify HA qualification support boundary

0c3dea0

Keep qualification evidence external

22ae0c0

Align HA qualification with initial scope

50534c4

Exercise paused stale command completion

9aff4dd

Bind HA qualification to release evidence

bd4e0c2

Make HA qualification evidence executable

ae2b972

Make power-loss qualification evidence explicit

cd54402

Close power-loss ownership intervals explicitly

cf1e386

Seed HA qualification ownership intervals

3d6276e

Bound HA qualification evidence uncertainty

e833ee0

Coordinate HA qualification ownership probes

883f004

Tighten HA qualification boundaries

c94bf46

State HA network trust boundary

7dd9d42

Scope HA qualification evidence

a99dbd7

Require durable command qualification evidence

00e38b3

Tighten command recovery qualification

3d7f819

Observe active work during HA qualification

4b3383a

Qualify HA effect exclusivity without polling claims

71f7e21

Bound post-failover curtailment restoration

20cd501

Keep HA qualification evidence gap-free

bb26db6

Qualify curtailment across every failover

b148810

Qualify curtailment during degraded operation

9bd4d3d

Make HA qualification claims conservative

337bf82

Qualify supported HA install platforms

f5e8144

Align HA qualification with simplified firewall

9dc0e10

Navigation Menu

Uh oh!

Define clean-install HA qualification - #889

Define clean-install HA qualification#889
ankitgoswami wants to merge 37 commits into
ankitg/ha-debian-installfrom
ankitg/ha-clean-install-qualification

ankitgoswami commented Aug 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ankitgoswami commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Areas of the code involved

Key technical decisions & trade-offs

Testing & validation

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔐 Codex Security Review

Review Summary

Findings

[MEDIUM] Qualification uses an unsupported installation flow

[MEDIUM] Failure gates do not isolate the fencing path being qualified

[MEDIUM] The no-overlap requirement contradicts the stalled plugin-call gate

Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ankitgoswami commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading